Using Semantic Overlap Scoring in Answering TREC Relationship Questions

A first step in answering complex questions, such as those in the “Relationship” task of the Text REtrieval Conference’s Question Answering track (TREC/QA), is finding passages likely to contain pieces of the answer—passage retrieval. We introduce semantic overlap scoring, a new passage retrieval algorithm that facilitates credit assignment for inexact matches between query and candidate answer. Our official submission ranked best among fully automatic systems, at 23% F-measure, while the best system, with manual input, reached 28%. We use our Nuggeteer tool to robustly evaluate each component of our Relationship system post hoc. Ablation studies show that semantic overlap scoring achieves significant performance improvements over a standard passage retrieval baseline.